Encoding and Decoding of Emotional Speech: a Cross-Cultural and Multimodal Study between Chinese and Japanese

Daily interactive communications involve the processing of verbal and nonverbal emotional cues from auditory and visual stimuli. The encoding and decoding schemes of emotional speech, however, have not been fully investigated in an interactive communication framework. Therefore, the present research is concerned with the encoding and decoding process of emotional speech during interactive communications based on an improved Brunswik's Lens Model. In this conceptual model, emotional speech is encoded by the distal features of the speakers, which are transmitted through multimodality transmission channels. It is decoded by the listeners, in which the emotional attributes are achieved to be the proximal percepts. The encoding and decoding of the emotion are modulated by the speaker and listener's language and cultural backgrounds, as well as the transmission channels. This hypothesis is taken as the basic consideration for the present study on multimodal emotional speech communication. In order to clarify the processing of emotional encoding and decoding in speech, three research issues have been examined through emotional speech perception, production and simulation for two typologically different languages, i.e., Chinese and Japanese. Specifically, (i) The decoding scheme is explored by carrying out four perceptual experiments in four transmission channels, i.e., vocal only (Aonly), facial only (Vonly), vocal & facial congruent (AVC) and vocal & facial conflicting (CAV). The study focuses on the linguistic and cultural effects on the perception of emotion attributes and proximal cues transmitted in multimodality channels; (ii) The encoding scheme is addressed by the analysis of the articulatory, segmental and suprasegmental acoustic features in both Japanese and Chinese; (iii) The relationship between encoding and decoding is checked by the simulation and perception of the expressive Chinese boundary tones. The approached major findings are as follows: (i) The listeners are able to recognize the portrayed emotions better than chance scores in three modalities, e.g., Aonly, Vonly and AVC. Additionally, they are able to recognize some common vocal and facial action features (e.g., proximal cues), which indicate that the emotional perception has cross-cultural psychological foundations. However, different speakers have their own encoding strategies. When facial expressions are